First-Order Newton-Type Estimator for Distributed Estimation and Inference

This paper studies distributed estimation and inference for a general statistical problem with convex loss that could be non-differentiable. For the purpose of efficient computation, we restrict ourselves to stochastic first-order optimization, which enjoys low per-iteration complexity. To motivate proposed method, first investigate theoretical properties straightforward Divide-and-Conquer Stochastic Gradient Descent (DC-SGD) approach. Our theory shows there is restriction on number machines this becomes more stringent when dimension $p$ large. overcome limitation, proposes new multi-round procedure approximates Newton step only using subgradient. The key component in our method proposal computationally estimator $\Sigma^{-1} w$, where $\Sigma$ population Hessian matrix $w$ any given vector. Instead estimating (or $\Sigma^{-1}$) usually requires second-order differentiability loss, First-Order Newton-type Estimator (FONE) directly estimates vector interest w$ as whole applicable non-differentiable losses. also facilitates empirical risk minimizer. It turns out term limiting covariance has form can estimated by FONE.